Papers with factuality metrics
Shortcomings of Question Answering Based Factuality Frameworks for Error Localization (2023.eacl-main)
Copied to clipboard
| Challenge: | Abstractive summarization systems often generate summaries with factual errors . many approaches to detect these errors have been proposed, but this capability has not been evaluated in past research . |
| Approach: | They propose to use question answering-based factuality metrics to detect errors in summaries . they find that QA-based frameworks fail to correctly identify error spans in generated summary . |
| Outcome: | The proposed methods outperform trivial exact match baselines in localizing errors in summaries. |
GO FIGURE: A Meta Evaluation of Factuality in Summarization (2021.findings-acl)
Copied to clipboard
| Challenge: | Recent advances in text generation systems produce fluent, coherent, relevant, and factually correct text. |
| Approach: | They propose a metaevaluation framework for evaluating factuality evaluation metrics . they propose five necessary conditions to evaluate factual metrics on diagnostic factuity data . |
| Outcome: | The proposed framework provides robust evaluation that is extensible to multiple types of factual consistency and standard generation metrics, including QA metrics. |
Trusting Your Evidence: Hallucinate Less with Context-aware Decoding (2024.naacl-short)
Copied to clipboard
| Challenge: | Language models (LMs) often struggle to pay enough attention to the input context, and generate texts that are unfaithful or contain hallucinations. |
| Approach: | They propose a context-aware decoding technique that amplifies the difference between the output probabilities when a model is used with and without context. |
| Outcome: | The proposed model significantly improves faithfulness of different LM families including OPT, GPT, LLaMA, and FLAN-T5 for summarization tasks. |
Evaluating the Tradeoff Between Abstractiveness and Factuality in Abstractive Summarization (2023.findings-eacl)
Copied to clipboard
| Challenge: | Abstractive summarization models generate fluent and well-formed output but lack semantic faithfulness, or factuality, with respect to the input documents. |
| Approach: | They propose new factuality metrics that adjust for the degree of abstractiveness . they propose to visualize the rates of change in factual as we gradually increase abstractiveity . |
| Outcome: | The proposed models generate fluent and well-formed summaries but lack semantic faithfulness, or factuality, with respect to the input documents. |
Merging Facts, Crafting Fallacies: Evaluating the Contradictory Nature of Aggregated Factual Claims in Long-Form Generations (2024.findings-acl)
Copied to clipboard
| Challenge: | Existing factuality metrics cannot evaluate paragraphs with ambiguous entities, authors show . |
| Approach: | They propose a new metric to evaluate the factuality of long-form generations from large language models. |
| Outcome: | The proposed metric can assess the factuality of people biographies with entity ambiguity better than FActScore. |
How Far are We from Robust Long Abstractive Summarization? (2022.emnlp-main)
Copied to clipboard
| Challenge: | Abstractive summarization has made tremendous progress in recent years . however, even under a short document setting, abstractive models often generate summaries that are repetitive, ungrammatical, and factually inconsistent with the source. |
| Approach: | They perform fine-grained human annotations to evaluate long document abstractive summarization systems and develop factual consistency metrics. |
| Outcome: | The proposed model can generate more relevant summaries but not factual ones. |
Are Factuality Checkers Reliable? Adversarial Meta-evaluation of Factuality in Summarization (2021.findings-emnlp)
Copied to clipboard
| Challenge: | Despite the progress of factual evaluation methods, they are limited in their opacity and lack the ability to assess the factuality of the summaries. |
| Approach: | They propose to use a meta-evaluation methodology to diagnose the fine-grained strengths and weaknesses of 6 existing top-performing metrics over 24 diagnostic test datasets. |
| Outcome: | The proposed method diagnoses the strengths and weaknesses of 6 existing top-performing metrics over 24 diagnostic test datasets and searches for directions for further improvement by data augmentation. |
Understanding Factuality in Abstractive Summarization with FRANK: A Benchmark for Factuality Metrics (2021.naacl-main)
Copied to clipboard
| Challenge: | Modern summarization models generate fluent but often factually unreliable outputs. |
| Approach: | They propose to use human annotations to identify different categories of factual errors and benchmark factuality metrics to improve summarization evaluation. |
| Outcome: | The proposed method identifies the proportion of different categories of factual errors and benchmarks their human judgements as well as their specific strengths and weaknesses. |
FactPICO: Factuality Evaluation for Plain Language Summarization of Medical Evidence (2024.acl-long)
Copied to clipboard
Sebastian Joseph, Lily Chen, Jan Trienes, Hannah Göke, Monika Coers, Wei Xu, Byron Wallace, Junyi Jessy Li
| Challenge: | FactPICO is a factuality benchmark for plain language summarization of medical texts describing randomized controlled trials . existing metrics for factual summarizing medical evidence are poorly correlated with expert judgments on the instance level. |
| Approach: | They propose a factuality benchmark for plain language summarization of medical texts . they assess factuality of critical elements of RCTs in those summaries . |
| Outcome: | The proposed benchmark assesses the factuality of medical summaries using LLMs . the summary summators are based on 345 plain language summaires with fine-grained evaluation . |
Understanding Factual Errors in Summarization: Errors, Summarizers, Datasets, Error Detectors (2023.acl-long)
Copied to clipboard
Liyan Tang, Tanya Goyal, Alex Fabbri, Philippe Laban, Jiacheng Xu, Semih Yavuz, Wojciech Kryscinski, Justin Rousseau, Greg Durrett
| Challenge: | Abstractive summarization systems still include factual errors in generated summaries despite recent improvements in factuality detection . |
| Approach: | They aggregate factuality error annotations from nine existing datasets and stratify them according to the underlying summarization model. |
| Outcome: | The proposed method improves on the ChatGPT-based model and shows that it is not superior for all error types. |
Evaluating and Improving Factuality in Multimodal Abstractive Summarization (2022.emnlp-main)
Copied to clipboard
| Challenge: | Current factuality metrics do not account for vision modality, thus are not adequate for vision-and-language summarization. |
| Approach: | They propose a weighted combination of CLIPScore and BERTScore to evaluate factuality for abstractive document summarization. |
| Outcome: | The proposed metric outperforms existing factuality metrics on four factuity metric-evaluation benchmarks and is robust to human judgments. |
Reference Matters: Benchmarking Factual Error Correction for Dialogue Summarization with Fine-grained Evaluation Framework (2023.acl-long)
Copied to clipboard
| Challenge: | Current evaluations of FEC models that depend on factuality metrics are not reliable and detailed enough. |
| Approach: | They propose a fine-grained evaluation framework that automatically evaluates FEC models on different error categories. |
| Outcome: | The proposed evaluation framework compares models on different error categories and finds the best training modes and significant differences in the performance of existing models. |
Optimising Factual Consistency in Summarisation via Preference Learning from Multiple Imperfect Metrics (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Existing evaluation metrics are unreliable for factual consistency tasks, limiting their effectiveness as signals for shaping model behaviour. |
| Approach: | They propose an automated training pipeline that improves factual consistency in summaries by aggregating scores from different weak metrics. |
| Outcome: | The proposed approach improves factual consistency in summaries by aggregating scores from weak metrics. |
Verify with Caution: The Pitfalls of Relying on Imperfect Factuality Metrics (2025.findings-acl)
Copied to clipboard
| Challenge: | Recent advances in large language models have led to optimism that they can serve as reliable evaluators of natural language outputs. |
| Approach: | They propose to use factuality metrics to evaluate natural language outputs . they find they misestimate the factual accuracy of NLG systems . |
| Outcome: | The proposed metrics are inconsistent with each other and often misestimate the factual accuracy of NLG systems, causing biases against paraphrased outputs and outputs that draw upon faraway parts of the source documents. |